Python 3.12 and other architectural improvements by axelstudios · Pull Request #5182 · SEED-platform/seed

axelstudios · 2026-03-18T21:36:11Z

What's this PR do?

Python v3.12+
Modernized code
Added uv with lockfile and pyproject.toml, removed requirements.txt files
Removed unused files
Enabled parallel test running via a custom backend that disables timescaledb background workers when cloning the test database for multiple workers
- All 1,322 tests run in <10 minutes locally with a 32-core machine, whereas it took ~50 minutes before
- CI tests have gone from over 2 hours, to 53-56 minutes
Switched to tox-uv
Switched to hypercorn for ASGI instead of WSGI
Dependency updates

How should this be manually tested?

Delete your existing virtual environment
Run uv sync
Run and test Django like normal
Run and test via hypercorn: uv run hypercorn -b 127.0.0.1:8000 --workers 4 config.asgi:seed

Run the tests locally via:

set DJANGO_SETTINGS_MODULE=config.settings.test
set SEED_PM_UN=XXX
set SEED_PM_PW=XXX
uv run python manage.py test --parallel=auto --verbosity=2 seed .

Modernized code Added uv, removed requirements Removed unused files

…oss the whole source test DB lifecycle instead of toggling it per clone

Copilot

Pull request overview

This PR upgrades the project to Python 3.12+, modernizes dependency management and CI (uv + lockfile + tox-uv), enables parallelized test execution, and migrates the runtime server stack from WSGI/uWSGI to ASGI/Hypercorn.

Changes:

Migrate Python dependency management to pyproject.toml/uv.lock (remove requirements*.txt) and update tox/CI to use uv.
Switch runtime + Docker/nginx wiring from uWSGI/WSGI to Hypercorn/ASGI.
Enable parallel Django tests via a custom PostGIS backend that mitigates TimescaleDB clone blocking; perform assorted Python 3.12 typing/stdlib cleanups and test fixes.

Reviewed changes

Copilot reviewed 92 out of 97 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tox.ini	Use uv dependency groups; route mypy via pre-commit.
seed/views/v3/media.py	Expand file-path matching to tolerate path normalization differences.
seed/views/v3/import_files.py	Centralize DST-aware localization via `time_utils`.
seed/views/v3/facilities_plan_run.py	Typing modernization for Python 3.12.
seed/views/main.py	Update celery app import; modernize path handling.
seed/utils/v4/inventory_filter.py	Use PEP 604 unions.
seed/utils/time_utils.py	Add localization helpers for DST edge cases.
seed/utils/search.py	Typing modernization; safer split usage.
seed/utils/labels.py	Add `default=[]` to `ArrayAgg` annotation.
seed/utils/inventory_filter.py	Typing modernization for Python 3.12.
seed/tests/test_search.py	Typing modernization in tests.
seed/tests/test_salesforce_views.py	Read Salesforce settings via `django.conf.settings`.
seed/tests/test_property_views.py	Use `localize_datetime` for DST test data.
seed/tests/test_portfoliomanager.py	Wrap bytes in `BytesIO` for `pandas.ExcelFile`.
seed/tests/test_permissions.py	Pytest-style exception assertions; cleanup ordering.
seed/tests/test_meter_views.py	Use `datetime.UTC` constant.
seed/tests/test_labels_api_views.py	Sort assignments for stable comparisons.
seed/tests/test_decorators.py	Pytest-style exception assertions.
seed/tests/test_columns.py	Make path-matching regex cross-platform.
seed/tests/test_certification_serializers.py	Pytest-style exception assertions.
seed/tests/test_audit_template.py	Use `datetime.UTC` constant.
seed/tests/test_api_mixins.py	Pytest-style exception assertions; cleanup ordering.
seed/tests/api/test_seed_host_api.py	Make API test runnable as module/script via dynamic imports.
seed/tests/api/test_modules.py	Make helper imports package-safe via dynamic imports.
seed/serializers/uniformat.py	Typing modernization.
seed/serializers/properties.py	Minor dict `.get()` cleanup.
seed/search.py	Remove unnecessary `__future__` import (Py3.12+).
seed/models/derived_columns.py	Remove unnecessary `__future__` import (Py3.12+).
seed/models/compliance_metrics.py	Typing modernization.
seed/models/columns.py	Typing import cleanup; PEP 604 unions.
seed/models/building_file.py	Minor `.get()` cleanup.
seed/models/analysis_property_views.py	Minor `.get()` cleanup.
seed/lib/uniformat/uniformat.py	Use stdlib `NotRequired` (Py3.12+).
seed/lib/superperms/orgs/permissions.py	Typing modernization.
seed/lib/progress_data/progress_data.py	Typing modernization.
seed/lib/mcm/reader.py	Regex raw string cleanup.
seed/data_importer/tests/test_meters_parser.py	Use `localize_datetime` for DST-related test data.
seed/data_importer/tests/integration/test_merge_duplicate_rows.py	Use `datetime.UTC` constant.
seed/data_importer/tasks.py	Defaultdict simplifications; stable UTC hashing for datetimes; typing modernization.
seed/data_importer/meters_parser.py	Centralize DST-aware localization via `time_utils`.
seed/data_importer/match.py	Typing modernization.
seed/celery_app.py	Remove script entrypoint; keep as importable celery app.
seed/backends/postgis_parallel_tests/creation.py	New DB creation logic to support parallel test DB cloning with TimescaleDB.
seed/backends/postgis_parallel_tests/base.py	New database backend wrapper for tests.
seed/backends/postgis_parallel_tests/init.py	Backend package marker.
seed/backends/init.py	Backends package marker.
seed/audit_template/audit_template.py	Avoid duplicate metering scenario computation.
seed/analysis_pipelines/utils.py	Defaultdict simplification.
seed/analysis_pipelines/eui.py	Ensure config date inputs become timezone-aware.
seed/init.py	Update celery app import path.
requirements/test.txt	Remove legacy requirements file.
requirements/local.txt	Remove legacy requirements file.
requirements/aws.txt	Remove legacy requirements file.
requirements.txt	Remove legacy requirements file.
pyproject.toml	New canonical dependency + tooling configuration.
package.json	Bump version; enforce pnpm usage.
ng_seed/views.py	Switch to pathlib + custom static file serving helper.
docs/source/setup_osx.rst	Update setup instructions for uv + hypercorn.
docs/source/setup_docker.rst	Update docker testing instructions and terminology.
docs/source/modules/config.rst	Document ASGI module instead of WSGI.
docs/source/migrations.rst	Update dependency install instructions to uv.
docs/source/linux.rst	Update Linux setup + server guidance for hypercorn/uv.
docs/source/developer_resources.rst	Update tooling commands to `uv run tox`.
docs/source/conf.py	Add type annotations to config values.
docs/source/aws.rst	Update AWS setup to uv + nginx/hypercorn.
docker/uwsgi.ini	Remove uWSGI config.
docker/supervisor-seed.conf	Switch supervisor program to Hypercorn.
docker/start_hypercorn_docker.sh	Add robust hypercorn startup with worker calculation.
docker/seed-entrypoint.sh	Update comment for web process user ownership.
docker/nginx/seed-uwsgi.conf	Remove uWSGI nginx include.
docker/nginx/seed-proxy.conf	New nginx proxy include for Hypercorn upstream.
docker/nginx/nginx.conf.template	Point upstream to Hypercorn; include proxy config.
docker/backup_k8s/Dockerfile	Install uv + awscli via uv tool; remove pip bootstrap.
docker-compose.dev.yml	Use Hypercorn with reload in dev compose.
config/wsgi.py	Remove WSGI entrypoint.
config/views.py	Add debug static/media serving views with caching headers.
config/urls.py	Wire debug static/media serving via custom views.
config/settings/test.py	Logging tweaks; test DB backend + in-memory cache for parallel tests.
config/settings/docker_test.py	New docker test settings for parallel-safe cloning.
config/settings/docker_dev.py	Simplify redis/celery config; remove “detect test run” branch.
config/settings/common.py	Switch to ASGI application; add swagger renderer setting; typing tweak.
config/asgi.py	New ASGI entrypoint + lifespan handling.
bin/start_uwsgi.sh	Remove uWSGI startup script.
bin/start_hypercorn.sh	New Hypercorn startup script using uv.
README.md	Update startup commands to Hypercorn.
Dockerfile.ecs	Switch Python install to uv + lockfile in ECS image.
Dockerfile-dev	Switch Python install to uv + lockfile in dev image.
Dockerfile	Switch Python install to uv + lockfile in prod image.
.spelling.dic	Remove uWSGI words from spelling dictionary.
.python-version	Set Python version to 3.12.
.pre-commit-config.yaml	Update hook versions; add mypy hook; rename ruff hook id.
.gitignore	Stop ignoring `.python-version`.
.github/workflows/ci.yml	Use uv in CI; run docker tests with parallel + docker_test settings.
.dockerignore	Ignore `.venv`.
.coveragerc	Reorder/adjust omit patterns.
.codeclimate.yml	Remove Code Climate config.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ng_seed/views.py

seed/backends/postgis_parallel_tests/creation.py

seed/analysis_pipelines/eui.py

pyproject.toml

Python v3.12

bb170dc

Modernized code Added uv, removed requirements Removed unused files

axelstudios added Maintenance Tag as maintenance if the issue relates to general cleanup, maintenance, etc. Do not delete label. Enhancement Add this label if functionality was generally improved but not a full feature or maintentance. Performance Bug labels Mar 18, 2026

axelstudios force-pushed the task/update-python branch 3 times, most recently from a1be08d to 4750ec3 Compare March 18, 2026 22:16

Implement test parallelization

3f6fdef

axelstudios force-pushed the task/update-python branch 2 times, most recently from 9a08b4d to dce7e7d Compare March 18, 2026 23:24

Attempt dedicated docker test config

d8e259f

axelstudios force-pushed the task/update-python branch 3 times, most recently from d06eeee to 91c08c1 Compare March 19, 2026 15:51

Updated postgis_parallel_tests to use the restore-mode workaround acr…

04716ae

…oss the whole source test DB lifecycle instead of toggling it per clone

axelstudios force-pushed the task/update-python branch from 91c08c1 to 04716ae Compare March 19, 2026 15:53

Cleanup, improved test logging

b98822c

axelstudios force-pushed the task/update-python branch from 6362cd3 to be617fd Compare March 19, 2026 20:06

Fix salesforce tests to run sequentially

41cd651

axelstudios force-pushed the task/update-python branch from be617fd to 41cd651 Compare March 19, 2026 20:34

axelstudios marked this pull request as ready for review March 19, 2026 21:36

Make salesforce tests parallel again

3edc7d3

axelstudios requested a review from Copilot March 19, 2026 21:37

Copilot started reviewing on behalf of axelstudios March 19, 2026 21:38 View session

Copilot AI reviewed Mar 19, 2026

View reviewed changes

ng_seed/views.py Outdated Show resolved Hide resolved

seed/backends/postgis_parallel_tests/creation.py Outdated Show resolved Hide resolved

seed/analysis_pipelines/eui.py Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

Cleanup

1475cc2

axelstudios requested a review from kflemin March 20, 2026 00:47

kflemin approved these changes Mar 26, 2026

View reviewed changes

Remove base.txt

bb63c04

axelstudios merged commit a02e3ca into develop Mar 26, 2026
9 checks passed

axelstudios deleted the task/update-python branch March 26, 2026 19:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.12 and other architectural improvements#5182

Python 3.12 and other architectural improvements#5182
axelstudios merged 9 commits intodevelopfrom
task/update-python

axelstudios commented Mar 18, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

axelstudios commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's this PR do?

How should this be manually tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

axelstudios commented Mar 18, 2026 •

edited

Loading